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***Titl e of th e Inv e ntion*** 

THREE-DIMENSIONAL SHAPE MEASURING METHOD AND ITS DEVICE 

** Summary of th e Inv e ntion** 

[0001] — A thr ee dim e nsional shap e is m e asured by a simpl e system structur e . A 

thr e e dimensional shap e measuring instrument compris e s a d e vic e (1 1) in which a light 
e mitting diod e (1 lb) is installed as a marker in a lin e laser light sourc e (1 la), an imaging 
d e vic e (1 2), and a comput e r (1 3). For m e asur e m e nt, a lin e las e r b e am from th e d e vic e (1 1) 
is appli e d to an obj e ct (1 4 ) to b e m e asur e d, th e imaging d e vic e (12) imag e s th e appli e d lin e 
las e r b e am (1 5) and th e light omitting diode, and a three dimensional shap e is obtain e d from 
th e imag e data by triangulation by mean s of th e computer (1 3). 
***** D e scription ******* 
**Fi e ld of th e Inv e ntion** 
Background of the Invention 

[0002] This invention is related to a^3D measurement system which use that uses 
laser-light and estimates depth information of an object by applying_a triangulation method; 
among these systems, this invention is especially related to a3D measurement system which 
that consists of a single camera to acquire accurate depth information. This invention also 
relates to a 3D measurement method using the 3D measurement system. 
**Background of the Invention** 

[0003] To acquire aJ3D shape of the object, an active stereo method based on the 
triangulation method is well known (pattern projection method and point projection method). 
These systems consist of light sources^ such as laser-lights or halogen lights,, and image 
capturing devices^ such as cameras or CCD(Charge-coupled Device) cameras. 3D shape 
estimation is conducted bya triangulation method of these devices. Therefore,_a precise 
relationship between the light source and the camera should be precalibrated, and thus, the 
whole system is prone to become large and complicated. 

[0004] If we can make the calibration process of the light source and the camera 
easier, an active 3D acquisition system becomes simpler. Based on such an idea, the-amethod 
which w as proposed that achieves easy calibration of the light source and the camera by 
attaching markers on the light source itself and capturing it with a single camera was 



"Low-cost interactive active monocular range finder". In Proceeding of Computer Vision and 
Pattern Recognition, 30-volume 1, pages 444-449, 1999.). 




Masahiro Takatsuka, Geoff A. W. West, Svetha Venkatesh,and Terry M. Caelli. 



2 

Summary of the Invention 

- [0005 1 H ow e v e r, sinc e b e caus e th ei rBecause the Takatsuka method assumes a beam 

of light for the light source, and thus, 3D information of only a single point can be acquired 
from a single image, it causes a serious problem feat -because it takes a very long time to 
capture 3D data of a large area. Another problem of their method is that, sinee-because 
markers attached to the light source are aligned along a single line, the p recision of acquired 
3D information is inevitably low. 

[0006] Thus, our invention aims to reduce the capturing time and improve the 
precision of 3D data of aJSD measurement system based on an_active stereo method in which 
a marker attached light source and a single camera are used. 
**Disclosure of the Invention** 

[0007] This invention achieves dense 3D data acquisition from a single image by 
using a line laser as the light source. With this feature, aJ3D shape can be efficiently acquired 
in a short time. In addition, because of the three-dimensional configuration of the markers, the 
light source position can be estimated accuratel y, and thus, . Thus, the precision of 3D data 
can be improved. 

[0008] Sinee -Because the p roposed method can acquire dense 3D points from a 
single image, if the acquired 3D points include 3D points already estimated in apast 
procedure, the p osition of the light source can be corrected by using the 3D points which that 
are already estimated , and thus, . Because of this, the accuracy of the 3D shape can be 
improved. 

**Briof D e scription of the Drawings** Brief Description of the Drawings 

[0009] The figure Figure 1 shows a configuration of fee-measuring devices ef-for 
image data as the-a_first embodiment of this invention. 

- [0010 1 Th e figur e Figure 2 shows a configuration of the measuring devices e£for 

image data as the-a^second embodiment of this invention. 

- [0011 ] Th e figur e Figure 3 shows an example of how to arrange the light-emitting 

diode(LED) markers, which is -are used in this invention. 

- [0012 1 The figure Figure 4 shows the relationship between the marker coordinate 

systen^ which areis fixed to the line-laser source^ and the camera coordinate system. 

- [00131 Th e figur e Figure 5 shows thea method for estimating the rotation of the 

marker coordinate system. 
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- [00141 Th e figur e Figure 6 shows thea method for estimating the positions of the 3D 

points used in this invention. 

- [00151 Th e figure Figure 7 shows thea method for estimating the difference between 

the marker plane and the laser plane. 

^^Description of th e Pr e f e rr e d Embodim e nts** Detailed Description of Preferred 
Embodiments 

[0016] In the following, an implementation of this invention is described, referring 
to the figures. First, we disclose the implementation of a3D measurement process using 
figure 1. At ajirst step, the users set up all objects in ord e r to so image capturing device(l-2) 
can capture both the light projecting device(l-l) and the target object(l-4) in a same frame. 
At the same time, an iris value and shutter speed are adjusted so that the laser light (1-5) 
projected onto the object and LEDs (1-lb) are accurately and simply detected by cameras by 
using an image processing method. In addition, the relationship between the light projecting 
device (1-1) and the sheet of light formed b y the line laser are measured in advance. In terms 
of an image capturing device, for example, we can use either avideo camera, a^CCD camera 
and so on. At the-a^second step, the users hold the light projecting device(l-l) by hand and 
project the laser onto the measured objec t (1-4) . Smee -Because the 3D configuration o f the 
LEDs are-is known, by detecting the positions of the LEDs on the image plane, the 
relationship between the light projecting device (1-1) and cam e ra the image capturing device 
(1-2) can be calculated. For calculation, anonlinear optimization are-is conducted to improve 
accuracy. 

[001 7] Siftee -Because the relationship between the light proj ecting device ( 1 - 1 ) and 
the laser plane is known, we can recovera depth of light projected area from the cam e ra 
image capturing device (1-2) with a triangulation method using the relationship between the 
light projecting device (1-1) and the camer a image capturing device (1-2) . 3D shapes of the 
target objects are obtained by repeating the process. If obtained 3D shapes are displayed on a 
display device (2-5) in real-time as shown in Fig._2, the users can learn that the area wher e 
dooG not m e asur e d vet on sit e of areas that have not vet been measured on site . Thus, fee-users 
can efficiently scan the 3D shape of the target object (2-4) . 

[0018] Examples ofa marker configuration are shown in Figure 3. I f the number of 
marker is more than 4,_a marker configuration is basically free; however, a,square-shaped 
configuration (3-1), an xyz coordinate z shaped configuration (3-2) A and acube^shaped 
configuration (3-3) have some advantage^ such as easy manufacture and simple calculation 
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for retrieving the relationship betwee n the light projecting device (1-1) and cam e r a image 
capturing device (1-2) . 

[0019] In the following, concr e t e a specfic implementation of a 3D measuring 
device and the method are described. 

[0020] Our system consists of a video camera and a laser-projecting device (Sea-see 
Fig.l). While measuring, the users hold the laser projector by hand and project the laser onto 
the measured object. The projected stripe is observed by the video camera. The shape of the 
target object is obtained by analyzing the video sequence. To estimate the position and 
orientation of the line laser-projecting device, LED markers are attached to the laser projector. 
LED markers are located so that they form a square and so that the square is placed on the 
same plane as the sheet of light. 

[0021] These markers define the local coordinate system in 3D space (4-1). We call 
this the "marker coordinate system" in this description. The origin of the marker coordinate 
system is placed at the center of the markers and we define the marker coordinate system by 
three unit vectors, el', e2' and e3\ The first and the second vectors, whose directions are 
expressed by el' and dl\ are taken so that they are parallel to the edges of the square formed 
by markers, (el* is parallel to the light projecting direction o f the laser projector and e2 f is 
perpendicular to fee -that direction.) The direction of the third vector e3 f is orthogonal to the 
plane formed by the four markers. We also define the camera coordinate system by el, e2 and 
e3, where el and e2 are respectively parallel to horizontal and vertical directions of the image 
plane, and e3 directed towards the backward direction of the camera(4-2). The center of the 
camera coordinate system is defined as the center of the lens. See Fig._4. 

[0022] Here, we name Stwo planes. The first one is the plane spanned by el 1 and 
e2', which we call the "marker planer"(5-4). The second one is the plane where the laser light 
is projected, which we call the "laser plane." The transformation between the "marker plane" 
and the "laser plane" aje-isrepresented by a 3D rotation matrix R and translation vector t. 
This transformation represents the marker positions from the cam e r a image capturing device . 
See Fig. 5. When the shape of the target object is scanned, the emitted light from the markers 
are-is_captured by the video camera. From the captured image frame, the locations of the four 
LEDs are obtained. In actual implementation, it can be achieved by simple thresholding and 
calculating the center of gravity of the connected pixels. By using the locations o f the LED 
markers, a description of the lines on the image which that g o through the markers is 
calculated and the crossing points of the lines are also calculated. The crossing points are 
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called a Focus of Expansion (FOE) Vanishing Points (VP) .(5-2a) and (5-2b). If the lines are 
parallel on the image, the FOE VP is defined as the point at infinity at the direction of the 
parallel lines. Let the 2D coordinates of these vanishing points be described as 
(Son > Sin X n = i n pixels, then, the 3D direction vectors(rl and r2) of those FOEs-VPs are 

r n ={gon>Su -/X (!) 
where f denotes focal length in pixel and 0/ denotes transpose of vectors and matrices. We 
define e\ = ^/[[^^(z = 1,2). The unit vectors e\ are used as an estimation of e\ . Estimation e\ 

is defined by taking a cross product of the two unit vectors as e\ = e[ x e\ . Figure 5 shows 
relationships between FOE VP and estimations £',(/ = 1,2). Using three unit vectors, 

estimated R of rotation matrix R can be described as: 

R = (e l9 e 2 ,e 3 ). (2) 
Let 2D coordinates of images of first and second markers be (p n , p l2 )> {p 2 \ > P22 ) • 3D 
locations in camera coordinates of the markers are 

m \ = (Pu U >Pl2 U >-f U y> m 2 = (P 2 \ V >P22 V -fry (3) 

where w, v are unknown variables. Let the 3D distance between those markers, which is 
known value, be Dm. Then 

H-rn 2 \\ = D m . (4) 
The equation of the marker plane is expressed as e' 3 [x + d = 0 , where x is a 3D point 
expressed in camera coordinates. Sinee -Because m l .m 2 are on the marker plane, 

e' 3 [m. +</ = 0(/ = l,2). (5) 

[0023] From equations (3)(4) and (5), d,u,v can be solved. Then we obtain i , as an 
estimation of / , by calculating the center of gravity of locations of the markers. 

[0024] To improve the estimation of the parameters of transformation, we optimize 
it with a nonlinear optimization. The evaluation function for optimization is defined as the 
sum of the squared distances between the actual LED positions on the image and the 
estimated marker positions on the image, which are calculated by projecting the LED position 
in local coordinates to the image plane by using estimated R and i . The function is 
described as 
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proj\x x , x 2 , x 3 )' ) = (O, )/ x 3 , (> 2 )/x 3 )' (7) 
where is a location o f the image of the / th marker, and m\ is a location of the z th 
detected marker in marker coordinates. proj() is an operation of projection from 3D position 
in camera coordinates into image coordinates. For actual implementation, we can use a 
simplex descending algorithm for the optimization. 

[0025] Then, using the estimated parameters of the marker plane, we can estimate 
the parameter of the laser plane. D e tail e d A detailed method will be described later. 
First, the pixels which that are projected by laser light (6-1) are extracted from the image (6- 
5). The pixels are easily picked up by asimple thresholding method, however, for actual 
implementation, skeletonizing and/or morphological filters are applied for improving to 
improve accuracy. The line going through the origin of the camera coordinates (6-4) and the 
surface point (line of sight) can be calculated for each pixel using intrinsic parameters, and 
the 3D location of the surface (6-3) can be determined by taking the intersection of the line of 
sight and the estimated laser plane(6-2). See-Fig.6 shows how to estimate surface locations 
by triangulation . 

[0026] Here, we explain the method to acquire acorrect relationship between the 
marker coordinates and the laser plane. The relationship can be defined as the parameters of 
the laser plane expressed in marker coordinates. The parameters can be expressed by (a, b, c, 
d) which represents_a plane in 3D space. If the markers plane is-precisely corresponds to the 
laser plane, the equation of the laser plane is z=0. 

[0027] For preparation of the calibration between the marker coordinates and the 
laser plane, the following measurement must be conducted in advance. First, a box-shaped 
object (7-5), of a known size a is captured. From the image, the extrinsic camera parameters 
are estimated using methods of camera calibration. From the extrinsic camera parameters and 
the size, the equations of the planes enveloping the box are estimated. 

[0028] We project the laser light onto the surface of the target object, and capture 
the lit surface and the markers. From the markers, the transformation between the marker 
coordinates and camera coordinates can be estimated by the p reviously described method. 
Then, the laser plane expressed in the camera coordinates can be calculated from the 
estimated laser plane parameters (a, b, c, d) expressed in marker coordinates. Next, the 
intersecting lines of the laser plane and the surface of the box shaped object are calculated. 
Then, the images of the lines can be determined (7-2). 
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[0029] We calculate the intersection of the laser plane estimated by the above 
method and the box-shaped object. Then, we estimate the image of the intersection and 
compare the image with the true positions of the image of the surface points lit by the laser 
(7-1). The squared distances between the pixels of the true image and the estimated image are 
summed up for all the pixels. The sum is the estimation function of the parameters (a, b, c, d\ 
which depicts the laser plane expressed in the marker coordinates. By minimizing the 
estimation function using a non-linear optimization method, we can estimate (a, b, c, d). The 
initial value for the optimization can be, for example, z=0, which is equivalent with 
(a,b,c,d)=(0,0,l,0). See figure 7. 

[0030] Finally, we describe a method to correct the estimation of the marker 
coordinate system from known 3D points. When the 3D locations of some part of the 
measured surface are known and if some part of the projected stripe of the laser is on the 
region of surface whose depth values are known, the known depth value and the depth value 
calculated from the estimated laser plane should match. Suppose we define an estimation 
function which that is the sum of function (6) and the sum of squared differences between the 
known depth value and the calculated depth value expressed above. The summation of the 
squared differences is calculated for all the region o f the surface whose depth values are 
known. Then, we can correct the estimation of the marker coordinate system by minimizing 
the estimation function. As for the known 3D points, we can use the 3D points precisely 
measured by other methods. Another example for the known 3D points is that we can select 
points which that are measured many times and the variances of whose positions are small. 
**Industrial usability** 

[0031] As explained above, applying this invention, we can quickly acquire a 
precise 3D shape without using complex equipment, which is conventionally required for 
active 3D measurement by triangulation. Moreover, the users can interactively check the 
regions whose positions are not measured while -during the measuring process by displaying 
the acquired 3D positions in real-time T ; thus, measurement of 3D shape of the target object 
becomes efficient. 
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Abstract of the Disclosure 
A three-dimensional shape is measured by a simple system structure. A three- 
dimensional shape measuring instrument comprises a device (1-1) in which a light-emitting 
diode (1-lb) is installed as a marker in a line laser light source (1-la), an imaging device (1- 
2), and a computer (1-3). For measurement, a line laser beam from the device (1-1) is applied 
to an object (1-4) to be measured, the imaging device (1-2) images the applied line laser beam 
(1-5) and the light-emitting diode, and a three-dimensional shape is obtained from the image 
data by triangulation by means of the computer (1-3). 



